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ABSTRACT 

The evaluation of "institutional effectiveness" is 
required by many accrediting agencies. The degree to which 
institutions have mobilized to meet the new criteria is the focus of 
this study. Surveys were sent to chief executive officers (CEOs) of 
institutions which will be reviewed within the next 5 years by the 
Southern Association of Colleges and Schools. Respondents were 
persons designated by the CEOs as having responsibility for the 
evaluation of institutional effectiveness, a total of 167 
institutions participated, for a response rate of 54%. This paper 
compares responses by: (1) the type of institutions (highest degree 
offered, governance, and size); (2) the function of the office 
responsible for evaluation; and (3) the existence of institutional 
research offices. Few differences were found across types of 
institutions. However, the overall results point out specific areas 
Vraere the majority of institutions are not currently engaged in 
ongoing, institutional assessment. An important finding of the survey 
is that the majority of institutions have not approached student 
outcomes assessment as an ongoing activity with an institution-wide 
perspective. The implications for the profession of institutional 
research are discussed. (Author/KM) 
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THE EVALUATION OF INSTITUTIONAL EFFECTIVENESS- 
THE RESPONSE OF COLLEGES AND UNIVERSITIES TO* 
REGIONAL ACCREDITATION 



Abstract 



The evaluation of "institutional effectiveness" is 
^-f e ?- by ° any acc "diting agencies. The degree to which 
institutions have mobilized to meet the new criteria is the 
focus of this study. Surveys were sent to chief executive 
officers (CEOs) of institutions which will be reviewed within 
the next five years by the Southern Association of Colleges 
and Schools. Respondents were persons designated by the CEOs 
as having responsibility for the evaluation of institutional 
effectiveness. 

i-^iil ° f mJ 67 institutions participated, for a response 
I . Paper com P ares responses by the type of 
institution: highest degree offered, governance, and size. 
Also, responses are compared by the function of the offic 
responsible for evaluation and the existence of institutional 
research offices. Fe- differences were found across types of 
institutions. Howe-er, the overall results point out 
specific areas where the majority of institutions are not 
currently engaged in ongoing, institutional assessment. The 
implications for the profession of institutional research are 
discussed. 
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INTRODUC HON 

R ^^v° Untabili ^ Y ^ h !? her educatio " become a common theme. 

Recently regional as well as specialized accrediting aaencies 
have established criteria which are concerned with outcomes 
assessment. The "institutional effectiveness" section of the 
Southern Association of Colleges and Schools' Criteria for 
Accredita tion (1987) focuses on the "evaluation of tf^T^uTts o"f" 
education and plans for the improvement of the institutional 
programs' 1 (SACS, 1987, p. 10). 

Although there has been much written and said about the 
evaluation of institutional effectiveness, it is not clear to 
what extent campuses have begun to mobilize efforts to meet these 
new standards. Pe ticularly institutions without central 
institutional research offices may lack the resources and skills 
to meet these new expectations. Moreover, small institutions, 
two-year colleges, and privately supported institutions may be 
less prepared for the demands of assessment than larger public 
four -year institutions. 

This study was undertaken to ascertain whether institution', 
are currently engaged in assessment activities rei-ted to the 
evaluation of institutional effectiveness. It also addresses the 
question of resources available to support assessment. 

LITERATURE REVIEW 

In ar article focusing primarily on the current state of 
evaluation knowledge regarding student testing, Baker and Herman 
(1985, p. 2) describe the changes in evaluation as a result of 
legislative action, social trends, and new technology; 

Simple linear models of evaluation, thought to mirror a 
linear p ttern of needs identification, planning, 
implementation, and evaluation, have been replaced bv 
analyses that recognize the complex interactions of 
technical, social, structural, and political environments. 
From simple, controlled studies of outcomes, design and data 
collection have been augmented to include studies of how 
policy goals, implementation and multifaceted information 
interact. Studies of evaluation have been enlarged to 
reflect a concern that the results be used by a range of 
decision makers (Baker and Herman, 1985, p. 2). 

This description closely reflects the changes in the 
accreditation process. Previously accr 3d i tat ion depended on 
linear models which assumed a high and positive correlation 
between inputs — the number cf volumes in the library, the percent 
of PhDs on the faculty, the student-faculty ratio — and the 
product of a quality institution. Now, however, accrediting 
agencies emphasize planning, evaluation and research as 
cont inuing and dynamic processes . 

The focus of accreditation has changed tremendously over 
the last decade. In the 1970s Troutt (1979), as cited in Feasley 
(1980, p. 28), conducted a content analysis of the published 
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criteria and standards of the regional accrediting associations. 
From this study five criteria that supposedly related to quality 
we*-e common to all accrediting agencies: institutional purposes 
and objectives, educational programs, financial resources* 
faculty, and the 1 ibrary/ learning center. However, he reported 
no research to support the relationship between these five 
r r i*?ri3 and institutional quality (Feasley* 1980, p. 28). 

The accrediting agencies have responded to the increasino 
demand for measures of quality, not by abandoning the traditional 
standards, but by adding new criteria that examine institutional 
effect iveness . Just as evaluat ion as a field has begun to 
abandon the li near mode 1 , so too for the accred i t i ng agenc ies . 
The new approaches are not yet crystal c 1 ear , but they are 
certainly more dynamic and more inclined to examine results. 
Accreditation is emphasizing the evaluation of institutional 
effect iveness through cont i nua 1 re-ex ami nat ion and reassessment . 
Thus, the focus of accreditation has moved from resources to 
results and outcomes (Council on Postsecondar y Accred l tat ion % 
1986, and Moore, 1986) . 

The content of this new orientation is changinG also. 
Lenni ng ( 1986 ) lists numerous "conceo tual ly muddy " areas in 
higher education where applied research is needed. He includes 
such areas as "unexpressed student needs; non-academic student 
development outcomes; and long-term consequences of college for 
students; . . .the strategic planning, managemei:t, and environmental 
scanning process; the development of measures and indicators of 
students' non-academic development" (Lenning, 1986, p. 5) — many of 
the same topics in which accreditation agencies are also 
interested . 

Lest we think that the new evaluation methodologies, the new 
content focus, and the emphasis on a dynamic process of self- 
evaluation are solely theoretical constructs, James Rogers, 
Execut i ve Di rector of the Southern Assoc iation of Col leges , and 
Schools (SACS), the regional accrediting association in the 
South* said in an address to the Twelfth National Conference on 
Blacks in Higher Education: "The whole thrust of this new 
emphasis Con institutional effectiveness! is to encourage 
institutions to engage in cont inuinq Cemphasis added! studv, 
analysis* and appraisal of their purposes, policies, procedures* 
and programs" (Rogers, 1987) . 

Indeed in a recent survey co-sponsored by the Education 
Commission of the States, American Association for Higher 
Education (AAHE) Assessment Forum, and the State Higher Education 
Executive Officers (SHEE)), Boyer et al. (1987) learned that two- 
thirds of all states have forma 1 assessment initiati ves . These 
ac t i ons range from encourag i ng i nst itutional ac t i on to statewi de 
moni tor ing and mandated statewide test ing . 

In a survey conducted by the American Council on Education 
(El-Khawas, 1987), 70'/. of the administrators reporting agreed 
that accrediting agencies should r equi re colleges and 
universities to demonstrate effectiveness. In an issue of the 
Association for Institutional Research Pr of essiona 1 Fi_le» Johnson 
and Christal (1985) recommend that "data collection and studies 
be done on an ongoing basis instead of once every several years 
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or in the crisis mode that usually accompanies most self-study cr 
accreditation deadlines" (p. 5). 

In add ition to accreditation pressures , Thrash ( .1987 ) 
says that "the identification and measurement of institutional 
ef f ec t i veness , the assessment of educat i ona 1 outcomes, or 
whatever terms we choose to call these exercises — have in recent 
years moved from being esoteric concepts, cui iosities and ooorly 
understood terms to becoming an integral part of institutional 
evaluation and planning" Cp.481). 

The assessment procedures required to evaluate institutional 
effectiveness is not without nost both in terms of time and 
monetary expend i tures . Much of the temporal cost is due to 
internal debates about which methodologies are most appropriate 
to examine which goals (Warren, 1983) and what it all means. As 
Moore (1986) puts it, "...it is necessary for Cadministrators3 
first to develop within the institution a collective 
understanding of the nature of the task and then to lead the 
institution through the steps requisite to its achievement," 
(p. 51) a process which might be characterized as pulling a camel 
through a hole in a microchip. 

Monetarily, we are just beginning to gc_t a glimpse of how 
expensive this new era of assessment will be. Ewell and Jones 
(undated paper) wisely caution against assuming astronomical 
costs when much of what is included in "assessment" is already 
being done on many campuses by testing centers, offices of 
institutional research, and academic planning offices. However, 
Ewell and Jones al*o point out that their cost estimates are 
approximate at best since they do not include many of the 
personnel costs associated with developing* administering, and 
analyzing locally-produced instruments nor coordination costs and 
other activities which existing offices presumably perform now. 

Further, as Boyer, et al. (1987) discovered, most states 
that have statewide assessment programs underestimate th<=> costs* 
particularly with regard to staff time.' As Brown (1986) points 
out, the evaluation of nonacademic units, which is part and 
parcel of the evaluation process, represents an additional demand 
on campus resources. 

In his presidential address to the Association for 
Institutional Research 1987 Forum, Don Reichard stressed the 
long-term 3nd far-reaching effects of the assessment of 
institut i ona 1 ef f ec t i veness : 

If we are to respond to the mandate to look increasingly 
at results, it will mean a reorder ing of priorities anri 
i nvestment in staff and systems . It a i so must mean 
collaboration, sharing, and joint development if we are 
to afford the needed systems and efforts (Reichard, ;9B . 



Whether willingly or not, once the commitment has been made 
to participate in an ongoing evaluation of the institution, 
col leges and universities must al 1 undergo a somewhat simi lar 
process. Their historical commitment to self-evaluation will 
determine whether they start from ground zero or slightly above. 



Moore (198S) describes three broad but critical steps which must 
be taken to examine the illusive "effectiveness" issue. She 
includes <1) the development of meaningful statements of 
institutional mission and goals* (2) the design and 
implementation of an institutional planning process* and (3) the 
ident if icat i on of indicators of effectiveness . 

The iaent if ication of criteria appropriate for judging 
institutional effectiveness is perhaps one of the most difficult 
tasks facing higher education. Fincher (1978) argues for 
criteria that focus on educational outcomes if the purposes and 
goals are stated in terms of student learning. Social* economic* 
and political effects are too far removed from the educational 
process to serve as valid criteria measures, according to Fincher 
(1978). Regardless of whether short-term or long-term effects 
are selected* administrators* faculty, and the public must agree 
upon criterion measures against which to judge the institutions 
success • 

The evaluation process has begun for many institutions* 
either because of their own wish to do so or because of external 
demands* or more likely because of a multitude of reasons. But 
where do institutions which must present evidence of 
i nst i tut i ona 1 ef f ec t i veness to an accred i t i ng agency stand with 
regard to the process? Are they at ground zero? Or are only 
some institutions at ground zero* and if so* which ones 7 What 
characterizes an institution which is wel 1— prepared to assess its 
ef fee t i veness? 

Whi le the new methodo logies and or ientat ion towards 
accreditation and evaluation are exciting in some quarters* 
institutions required to carry out the new mandates may describe 
the process in terms other than "challenging." 

The present study attempts to shed light on the question of 
where the institutions are vi s-a— vis the new eva luat i on mode . 
Institutions accredited by the Southern Association for Colleges 
and Schools (SACS) constituted the sample for the study* but the 
findings should be useful across the LLS. as the demands of 
accreditation change from an examination of resources to one of 
outcomes and effect i veness . 

Spec if ico 1 ly the research addresses the extent to 
which institut.^ns that will be reaccredited within the next 
five years by SACS are currently assessing institutional 
effectiveness. Differences in institutions' involvement in 
assessment activities and in the support for the office 
responsible for assessment are compared for public versus private 
institutions* by the level of institution and by high versus low 
enro 1 lment • 

METHODOLOGY 

SACS provided the following information for the study: neme 
of institution, address* chief executive officer* initial and 
reaffirmation dates of accreditation* governance (publir vs. 
private)* level (I, II, III* IV), and enrollment. Level I 
institutions are community and technical colleges; Level II 
institutions offer four-year degrees only; Level III institutions 



are comprehensive colleges and universities, offering master's 
and professional degrees; and Level IV institutions are doctoral- 
granting uni versi t ies ♦ 

Enrollment is divided into hi^h and low enrollment based on the 
median enrollment of participating institutions. 

Institutions that will be reaccredi ted within the next f ; ve 
years constituted the sample. A total of 311 institutions met 
this criteria, representing 50% of the total. 

The worksheet for the required statements of the Cr i teria 
*fo r Accreditation: Commission on Co 1 leges (Southern Association 
of Colleges and Schools, 1987) served as the basis for the 
first 31 items on the questionnaire. These items describe 
evaluation, planning, and institutional research activities 
required to comply with Section III, Institutional Effectiveness 
Criteria. The wording of the items closely resembles the 
sentences on the worksheet. For each activity respondents 
indicate if it were "performed systematically and campus-wide*" 
"either not systematic or ...not campus-wide, " "unknown," or "not 
applicable." If the activity were performed on a regular, 
campus-wide basis, then the respondents indicate the extent to 
which their offices carried out this activity. The resDcnse 
CaXBQor i e= are "not at all? 1 * "somewhat," "to a large extent," and 
"como let el v responsible. " 

The next part of the questionnaire pertains to the resources 
available to the respondents' offices. First the respondents 
assess the adequacy of the budget, the size of the staff, and the 
knowledge, skills and experience of the staff to cerrv out 
activities for which their offices are currently responsible. 
Then they assess the adequacy of resources for carrying out all 
the evaluation, planning, and research activities described in 
the first set of items. Ea^h questijn has five possible 
responses, ranging from "strongly agree" to "strongly disagree." 

Three experts in institutional research who were very 
familiar with the Cr i ter la reviewed the instrument and offered 
suggestions that were incorporated when possible. Approximately 
ten institutional research and planning directors comp leted the 
final draft of the survey as a pert of the pretest. 

The chief executive officer received a letter with the 
questionnaire attached. The letter asked that he/she direct the 
qupst ionnaire to "the person who will provide the data and 
analysis to support the evaluation process" required for the 
insti tut ion's accreditation. The letter stated that in many 
cases the person will be the director of insti tutional research , 
but that persons with other ti ties may be assigned this 
responsibi 1 i ty . 

The Institute of Higher Education at the University of 
Georgia printed and mailed the surveys in the winter of 1987. 
Chief executive officers returned a postcard indicating that 
their institution would participate in the survey and gi v ing 
the name of the individual who would respond to the 
questionnaire. Chief executive officers who had not returned the 
postcard received a follow-up mailing. Several weeks after the 
due date of February 27, 1987, we identified institutions that 
had agreed to participate but had not returned the questionnaire. 
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We- telephoned the person whom the chief executive officer had 
indicated would respond to the survey to encourage participation* 
The persons whom the chief executive officers designated as 
responsible for the research and data analysis completed the 
surveys, including information about their titles and the offices 
to which they report. Also • -gani zat ional charts were returned 
with the surveys by approximately 43Vi. 

Two derived variables are the function of the? respondent's 
office and the existence of an institutional research office. 
The function of the office was coded based on the respondent's 
title* witn titles designating responsibilities for research* 
evaluation, and/or planning contituting one category and all 
other titles grouped into another category. The existerce of an 
institutional research off i ;e was based on three sources of 
information. I** the title of the person respond i ng was 
dit ector/coordina^:or of institutional research, we concluded that 
such an office existed. Of those institutions that sent 
org an i zat ional char ts , the listing of an institutional research 
or research/evaluation office led to the conclusion that such an 
office existed. In the absence of other information, the HEP 37 
Hi gher Educ ation Directory (Higher Education Publications* 1987) 
was used. A code of 09 indicated that the college had an 
institut ional research office . 

After the coding of variables, the data were entered into a 
data set, and SAS was used for both file manipulation and data 
analysis. 

The relationship of governance, level, and enrollment to the 
first 31 items is tested using ^ chi square analysis. Resoonses 
of "not applicable" were collapsed with "unknown" and "activity 
not systematic or not campus-wide" except for six 

items which were judged inappropriate for some institutions. For 
these six items, responses of M not aoplicable" were omitted from 
the analyses. Further analysis explored the relationship of 
research, planning, and evaluation activities to the existence of 
an institutional research office and the function of the office 
given the assignment of carrying out the evaluation of 
institutional effectiveness. Because of the large number of 
i terns , a conservat i ve alpha of .01 is used . 

The items relating to resources and scale scores, created by 
summing across variables, are analyzed using the general linear 
fliodel ( GLIi> . An alpha of .05 is used in these analyses. 

For scale scores, responses of "systematic and camous-wide" 
were scored as 2, "not systemat i c or not campus-wide" as 1 , and 
"unknown" or "not applicable" (when judged applicable to all 
institutions) as 0. The institutional effectiveness scale is the 
sum of the 31 items. This scale has a reliability coefficient 
for internal consistency of .88. The three subscales are 
Evaluation, with 18 items and a reliability coefficient of .77; 
Planning, with 4 items, and a reliability coefficient of .65; and 
Institutional Research, with 9 items and a reliability 
coefficient of .76. 



RESULTS 



A total of 167 institutions participated in the studv* for a 
response rate of 547.. The institutions who participated in the 
^ th£ / rppre=ant^t i v'£ fit tfts institutions tl~<*t will k= 

reaccredited within the next five years. Institutions that 
responded to the survey do not di f f er significantly from those 
that did not respond when compared by state? governance* level, 
and enrollment. 

The majority that participated in the study were public 
institutions (657.) . Approximately half (467.) were Level I 
institutions; 217., Level II; 207., Level III, and 137., Level IV. 
Of the 108 public institutions, the majority (607.) are Level I 
institutions. In contrast, of the 59 private i nst i tut ions , 517. 
are Level II institutions. 

Sixty-one percent of the 167 inst itut ions do not have 
institutional research offices. A slightly hiaher proportion of 
private as compared to public institutions do not have 
institutional research offices (657. versus 577.). Only 417. of the 
respondents work in an office of research* evaluation* or 
planning. The range of respondents' titles is extensive* from 
chief executive officer to faculty serving as chair of the self- 
study commi ttee . 

In Table 1 are the percentages of institutions reporting that 
the evaluation, planning, and institutional research activities 
are currently performed at least every five years on a campus- 
wide basis. Percentages for all respondents as well as 
public and private institutions are presented. 

Over 907. report that the evaluation of full-time faculty is 
a regular, campus-wide activity. Seventy-two percent indicate that 
they have estaolished guidelines for the use of faculty evaluations 
and procedures for planning and evaluation. Two-thirds have 
defined a process for curricular planning, review, and 
evaluation. Seventy percent carry out evaluations of part-time 
faculty and of the curriculum. Other activities that the 
majority of institutions do on a regular, campus-wide basis are 
the development of plans for facilities (697.), faculty workload 
studies (667. ) , evaluat ions of library (61V.), st ud ies of the 
effectiveness of instruction (597.), the development of plans for 
maintenance of property (577.), the development of plans for 
faculty assignments (567.), evaluations of administrators (567.), 
linkage of planning to the budget process (527.), and the 
evaluation of student development services <507»). 

Student outcomes are defined by 417., and 447. carry out 
evaluations of student achievement and outcomes. One third of 
respondents currently evaluate institutional research and show 
the relationship of institutional research to planning and 
evaluation. 

Of those institutions whosa mission includes public service 
mission, 477. develop goals for continuing education, 487. conduct 
evaluations of off-campus programs, 30V. document the 
effectiveness of continuing education, and 237. evaluate the 
publ ic service mission. For inst itutions which have graduate 
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teaching assistants, A 6'/. evaluate their effectiveness. For 
institutions Wlt h a research mission, only 33'/. evaluate 
n» 1 5b ion. 

in^IT * naly5i5 , fa y governance (public versus private 
institution), level, and enrollment yielded few significant 
differences. Because of f.xe large number of chi square enalvses, 
an alpha level of .01 was used. ' 

Public and private institutions had significantly different 
responses on only three items. Proportionately more public than 
private institutions evaluate the effectiveness of the oublic 
th™ CB K m K 1551 ° n (30,/ ' vs - »*> -^n responses were limited to 
those who have a public service mission. A larger percentage of 
v= B °^ll C K in5tltutions compared to private institutions' (66% 
vs. 41/.) have a maintenance plan for the uokeep of their 
property. Finally proportionately more public than private 
institutions (80'/. vs. 50V.) develop a facilities plan for the 
entire campus. 

The level of the institution was statistically siooificant 
Inn r rB H ra5 f 5 ' , Almost two-thirds (65'/.) of Level I (c.mmunity 
and technical colleges) institutions report that evaluations of 
off-campus programs are carried out, as compared to about one- 
third of Leve II and Level III and 19'/. of Level IV institutions. 
Proportionately m0re Level I institutions (86%) conduct 
evaluations of part-time faculty, whereas 66'/. of Level II, 63'/. of 
Leve. Ill, and 50% of Level IV institutions systems t i ca 1 1 v 
evaluate part-time faculty. Proportionately fewer Level IV 
(doctoral-granting) institutions (71%) evaluate full-time faculty 
T n ? T^i^: t:am P U5 - w ide basis, as compared to Level I (94%), 
Level II (91%), and Level III (97%) institutions. 

Student enrollment related to four activities. Larger 
institutions were more likely to report the establishment of 
procedures for institutional planning and evaluation (81% vs. 
64/.) and the development of a facilities plan (81% vs. 57%) than 
were smaller institutions. Institutions with high student 
enrollment we re also more likely to report evaluatino 
institutional research activities (44% vs. £37. ) and to 
demonstrate that institutional research supports planning and 
evaluation (45% vs. 23%). 

Not surprisingly, i stitutions with institutional 
research offices were more likeiy to report systematic, campus- 
W ,a«V V ln evaluatin °. institutional research activities (5i% 
vs. 3SA) and in demonstrating that institutional research support* 
planning and evaluation (45% vs. 25%). However, the existence of 
institutional research offices did r.ot relate significantly at 
the .01 level to any of the other items. 

If the function of the office assigned responsibility for 
the evaluation of institutional effectiveness was research, 
p.anmng, and/or evaluation, then the institutions were more 
likely to be involved in these campus-wide activities: 
the establishment of procedures for institutional planninq and 
evaluation (86% vs . 63% ); research on institutional purposes, 
policies, procedures, and programs (64V. vs. 36%); the evaluation 
of institutional research activities (52% vs. 21%); and the 
demonstration that institutional research supports planning and 
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evaluation (53V, vs. 2!*/.). 

The responses to questions relating to resources varied 
greatly depending or, whether the question referred to the 
office's present responsibilities or all the activities reouired 
to assess 1 nst \ tut i ona 1 effectiveness. The majority (55V.) 
indicate that their budget is adequate to carry out present 
responsibilities, whereas 77V. state that the budget is not large 
enough to assume all the planning, evaluation, and research 
activities mandated by Southern Association of Colleges and 
Schools. However, 557. state that the staff is not large" enouoh 
to carry out current responsibilities; and 81X, that their staff 
is inadequate for all activities pertaining to the evaluation of 
institutional effectiveness. Whereas two-thirds state that their 
starf have sufficient knowledge, skill, and experience for 
performing their current jobs, 55'/. say that their staff does not 
have the expertise to assume responsibility for the evaluation of 
institutional effectiveness. (Refer to Table 3.) 

Means by governance, level, and student enrollment were not 
statistically different. The existence of an institutional 
research office related to this item: "My staff is large enouoh 
to carry out all activities listed above." Institutions with 
institutional research offices were less likely to agree with 
this statement than those without institutional research offices 
(1.63 vs. 2.01). Respondents who were in offices responsible for 
research, evaluation and/or planning were also less likely to 
agree with this statement than those whose offices had other 
functions (1.66 vs. 2.01). 

DISCUSSION AND IMPLICATIONS 

These results, in general, did not identify major differences 
across types of institutions in the assessment of institutional 
effectiveness. The few differences that emerged were not 
surprising . 

An important finding of this survey are that the majority 
of institutions have not approached student outcomes assessment 
as an ongoing activity with an institution-wide perspective. 
This Picture confirms Secretary Bennett's view that traditionally 
institutions of higher education have focused on the evaluation 
of inputs— resources such as facilities, faculty, library books — 
and have not attempted to assess outputs— what students have 
learned as a result of attending college <Ihe Raleigh News and 
Obs erver , p. 22A) . The new emphasis by accrediting agencies "on 
the assessment of outcomes will pose challenges for institutions. 
The definition of expected outcomes, currently done by of the 

institutions in the sample, is the first step toward the 
development of assessment procedures. Only a third have 
recommended or selected ways of evaluating the achievement of 
educational outcomes. Within the next few years these 
institutions must demonstrate that they are evaluating the 
achievement of student outcomes and are usina this information in 
program improvement. Ce-tainly the data suggest that the 
assessment of student outcomes is an area of weakness among many 
institutions, regardless of size, governance, and level. 
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AM institutions must designate an administrative unit 
responsible ' institutional research and have a process for 
evaluating r f ec t i veness . However, in this sample onlv 39V. 

have formal .institutional research offices. Only of the 

offices responding to the survey function as research, planning, 
and /or evaluation offices. Before accreditation, 

responsibilities for planning and evaluation must be 
administratively assigned. Furthermore, institutions will have 
to develop procedures for evaluating the effectiveness of 
institutional research. The results suaaest that many new 
organizational units with institutional research responsibilities 
will emerge in the next few years. 

From these results we conclude that institutions have been 
less concerned about the assessment of research and oublir 
service activities than those directly related to instruction in 
degree-granting program? Other areas that will receive emphasis 
in the next few year- are the evaluation of the research mission; 
continuing education, extension work, and the public service 
mission. 

The offices which are responsible for the analytical support 
for the accreditation process indicate that thev need more staff 
in o~der to carry out their present activities. To assume 
responsibility for all activities related to the assessment of 
institutional effectiveness, the offices will need laroer 
gets, more staff, and staff with more expertise. Overall, 
nese findings are consistent across different types of 
institutions. This picture indicates that institutions with 

broad missions of teaching, research, and public service must 
engage in more new assessment activities than institutions with 
the more specific mission of undergraduate teaching. The survey 
does not provide answers about the extent to which institutions 
are planning for these assessments. A follow-up survey of the 
same institutions, after they have undergone accreditation, would 
provide information at >ut changes in research, planninq, and 
evaluation activities. If the new Criteria accomplish the 
intent, we expect that the assessment of institutional 
effectiveness will be a continuous process. We also expect that 
information gained from the assessment procedure will be used in 
institutiona planning and result in discernible changes. 

It is somewhat heartening tr learn that all institutions 
are doing something to prepare for accreditation. Unfortunately, 
not all are doing enough — far from it. The disconcerting fact is 
that less than half of the institutions report evaluation efforts 
in 16 of the 31 areas, Contrary to expectation, no institutional 
characteristic distinguishes those who are actively engaged in 
continuous, campus-wide assessment from those who are lf?ss 
involved in institutional assessment. Neither size, nor level, 
nor governance has strong relationships to the evaluation of 
inst i tut ional effectiveness . 

A disturbing finding is that the presence of an 
institutional research office has little effect cn assessment 
activities. Perhaps Pace's description of the dilemma of 
institutional research offices provides us with insight into the 



reasons : 

One might suppose that the institutional research 
office within the college would be the natural locus for 
ongoing institutional case study. But it might not 
be. Most such offices are beset by deadlines and 
heavily involved in basic accounting activities related 
to budget making, cost analysis, and similar matters, 
all of which orient the staff and its activities to serve 
administration and management ... .A case study needs data* 
in large amounts, but it also requires time for exploration, 
for reflection, and for thoughtful evaluation. Some insti- 
utional research offices have the capacity for educational 
evaluation as well as institutional accounting. Some do not 
(Pace, 1979, p. 124). 

Not surprising is that those serving in a research capacity 
were most likely to indicate insufficient staff resources to 
carry out the full array of evaluation and planning activities. 
They, better than anyone, know the dilemma described by Pace and 
the extent of work required to conduct quality evaluations. 

These findings lead to i ^commendations for all 
institutions — regardless of size, public or private, doctoral or 
two-year . 

RECOMMENDATIONS 

First all institutions should designate an administrative 
unit responsible for institutional research activities. In this 
sample, only 39% have institutional research offices and only 427. 
of those responding to the survey function, at least in part, as 
research, planning and/or evaluation offices. Clearly defining 
the roles of the office in the assessment of institutional 
effectiveness should assist in eliminating activities that 
detract from well-thought out assessment. 

Second the research office must be sufficiently staffed so 
that it can carry out the "case study" described by Pace (1979), 
a necessary activity to satisfy accreditation demands. Across 
institutions, our re.— indents indicate the need for more staff 
just to carry out present activitifc-. To assume responsibility 
for all assessment activities, the offices will need larger 
budgets, more staff, and more experienced, knowledgeable staff. 
As Pace (1979) suggested nearly a decade ago, technical expertise 
is often the selection criteria for a director of institutional 
research. However, to carry out an "institutional case study" 
requires a much broader knowledge of higher education, teaching 
and learning, student development, and organizational dynamics 
than specialized skill in data systems. Similar concerns were 
expressed by Van Maanen (1987) in his keynote address to the 
Association for Institutional Research, in which he called for 
institutional researc/ers to expand their horizons beyond the 
tabulation of data and into a context of assessment. 

The survey results indicate a need for more staff and a more 
highly knowledgeable staff to carry out all assessment 
activities. Thus while an expansion of staff is critical to 
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fulfilling accreditation requirements, the qualifications of the 
staff m^y be more important. Experience in program evaluation 
and testing is more critical now than in the past. 
«.k * Third ' colle 9e and university administrators must recognize 
that costs are associated with the new accreditation 
requirements. The costs cannot simply be absorbed by existina 
units, both because of the need for well-trained researchers but 
also because few institutions are currently carrying out all th<= 
required activities. For instance, institutions have to dev»loo" 
procedures for evaluating the effectiveness of institutional 
research. their public service mission, and their research 
activities. The area of student outcomes research alone can cost 
thousands of dollars, as institutions either purchase existinq 
instruments or undertake the complex and expensive process of 
locally developing and validating instruments. Institutions can 
not simply add on to these tasks without increasino office 
budgets substantially. 

It was hoped that the survey would sucgest an organizational 
structure for supporting assessment activities. However, the 
questionnaire data do not indicate that the oroani zat ional 
placement of the office responsible for analvtical "support is 
critical. Future research will focus on the outcomes of the 
accreditation process. For instance, are institutions indicating 
systematic, campus-wide research, evaluation, and plannino 
activities better prepared for their reviews? Does the evaluation 
of institutional effectiveness lead to changes in *- h P 
institution? Is the review process aided by the existence of an 
institutional research office? Where ar . the new research 
offices, established in response to the SACS criteria, located in 
the organization and what responsibilities are assigned to these 
offices? What kind of on-going i ns c i tut i ona 1 support is given to 
evaluation activities? It is expected that new organizational 
structures will appear to enable institutions to respond to the 
demands for accountability. 



Table 1 



Percent of Institutions Carrying Out Activity 
on a Systematic, Campus-Wide Basis 

ALL PUBLIC PRIVATE 

Conduct evaluations of full-time ~~ 

faculty 90mQ 90 ^ 91 ^ 5 

Establ ish guidel ines for the use 

of f acul t> evaluations 72 .4 72 . 1 72 .9 

Establish procedures for institu- 

t iona 1 p lann i ng and eva luat i on 72 .4 76 .9 64 .4 

Conduct evaluations of part-time 

facult Y 69.9 72.1 66.1 

Conduct evaluations of curricula 69.8 69.9 69.5 

Develop a facilities plan for the 

institution 6 g. 9 79#6 5Q#0 

Define a process for curricular 

planning, review, and evaluation 66.9 72,1 57.6 

Conduct studies of faculty work- 
loa d 65.6 



68.3 61.0 



Conduct evaluations of library 

services and programs 60.7 63.5 55.9 

Conduct studies of the effective- 
ness of instruction 59.3 57#3 62#7 

Develop a maintenance plan for 

upkeep of institutional property 57.1 66.0 41.4 

Conduct evaluations of administra- 
te* 56.4 62.5 45.8 

Develop a plan f*;r the assignment 

of faculty responsibilities 56.2 57.7 53.5 

Demonstrate that educational plan- 
ning guides budget preparation 5K9 55.8 44.8 

Conduct evaluations of student 

development services 49.7 51 .9 45.8 

Conduct evaluations of off-campus 

programs 4 7#8 53# ^ aQQ 
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Table 1 continued 



Percent cf Institutions Carryina Out Activity 
on a Systemat ic , Campus-Wide Basis 

ALL PUBLIC PRIVATE 

Carry out research studies of 

institutional pur poses , etc . 47.5 50.0 43.1 

Deve 1 op go a 1 s for cont i nui ng edu- 
cation and extension 47.1 49.0 41,7 

Conduct evaluations of ad missions 

policies 46.9 45.6 ^9.2 

Conduc t eva 1 ua t ions of gr adua te 

teaching assistants 46.2 45.5 50.0 

Conduc t evaluations of student 

outcomes 43. 6 ^ 6 m 2 39-0 

Document that evaluations are used 

to improve teaching 42.6 44.7 39.0 

Define expected student outcomes 40.5 44.2 33.9 



Conduct evaluations of the safety 

plan 37 # 7 



43.3 27.6 



Demonstrate that institutional 

research supports planning , etc . 33.7 39.4 23.7 

Evaluate institutional research 33 . 1 36 . 5 27 . 1 

Recommend me thods of educa t i onal 

assessment 33. ! 33.7 32#2 

Evaluate institution's research 

mission 33.0 37.1 23.1 

Document ef f ec t i veness of continu- 
ing education, extension, etc. 29.8 33.3 20.5 

Document that evaluation of students 
discriminates high and low achieve- 

ment 28.8 29.4 27.6 



Evaluate public service mission 23.4 29.6 

lit 
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Table 2 



ERIC 



Percent of Institutions Carrying Out Activity 
on a Systematic, Campus-Wide Bas?s 3y Level 



LEVEL 
I II III IV 



Conduct evaluations of full-time 
faculty 



94 91 97 71 



Establish guidelines for the use 

of faculty evaluations 75 ^5 7^ 71 

Establish procedures for institu- 
tional planni ng and evaluation 70 59 87 gi 

Conduct evaluations of part-time 
f acul ty 



Conduct evaluations of curricula 



84 62 61 43 

69 71 77 62 
Develop a facilities plan for the 

institution 7Q 71 6? 

Define a process for curricular 

planning, review, and evaluation 71 56 71 62 
Conduct studies of faculty work- 

load 69 59 61 71 

Conduct evaluations of library 

serv i ces and programs 69 44 5g 62 

Conduct studies of the effective- 
ness of instruction 65 56 57 ^ g 

Develop a maintenance plan for 

upkeep of institutional property 67 36 55 57 
Conduct evaluations of administra- 

tor5 66 32 61 52 

Develop a plan for the assignment 

of faculty responsibilities 62 4g 55 ^g 

Demonstrate that educational plan- 
ning guides budget preparation 56 42 5g 43 

Conduct evaluations of student 

development services 55 ^ ^g ^g 
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Table 2 continued 



Percent of Institutions Carrying Out Activity 
on a Systematic, Campus-Wide Basis by Level 



LEVEL 



I II III IV 

Conduct evaluations of off-campus 

programs 65 33 33 19 

Carry out research studies of 

institutional pur poses , etc . 43 33 65 62 

Develop goals for continuing edu- 
cation and extension 54 27 48 45 



Conduct evaluations of admissions 

po 1 ic i es ^7 



38 57 48 



Conduct evaluations of graduate 

teaching assistants O 100 57 35 

Conduct evaluations of student 

outcomes ^9 M 23 33 

Document that evaluations are used 

to improve teaching 46 47 35 33 

Define expected student outcomes 49 41 23 33 
Conduct evaluations of the safety 

plan 40 27 k2 3Q 

Demonstrate that institutional 

research supports planning , etc . 375 21 35 48 

Evaluate institutional research 30 26 45 38 
Recommend methods of educational 

assessment 40 26 23 33 

Evaluate institution's research 

mission 35 13 3 0 47 

Document effectiveness of continu- 
ing educat ion, extension, etc . 32 25 25 35 

Document that evaluation of students 
discriminates high and low achieve- 
ment 32 33 17 29 

Evaluate public service mission 25 12 17 43 
0 16 
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Table 3 

hi^repiTicut w i tn St^teiVie-nts Pc 1 at i ng to Office Resources 



Percentage of Respondents 
SD D u A SA 

The budget for o»-/ office is 
adeouate to carry out activities 

for which my office is responsible 10 26 9 50 5 



My staff is large enough to carry 
out the activities for which my 

office is responsible. 17 38 9 34 2 

My staff has the knowledge, skills* 
and experience to carry out the 
activities for which my office 

is responsible. 2 21 10 48 19 



The budget for my office is 
adequate to carry out all 

activities listed. 37 41 8 14 <1 

My staff is large enough to 

carry out all activities listed. 42 39 10 8 <1 

My staff has the knowledge, skills, 
and experience to carry out all the 

activities listed. 21 35 12 26 6 



SD = Strongly disagree 
D = Disagree 
U = Undecided 
A = Agree 

SA = Stronglv agree 
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Table 4 

Means of Planning, Evaluation, Institutional Research, and 
Institutional Effectiveness Scales 



EVAL PLAN IR TOTAL 

Governance 

Public 19.47 6.11 12.00 37. 5*, 

Private 18.63 5.64 10.56 34.67 

Level 

I 20.01 6.17 11. SO 37.89 

II 17.81 5.71 9.93. 33.10 

III 19.03 5.84 12.16 37.13 
Iv IB. 38 5.67 11.67 35.71 

Enro 1 1 ment 

Hi 9 n 19.05 6.13 12.20 37.32 

Low 19.28 5.77 10.80 35.76 

IR Office 

Y * 5 18.08 6.03 11.76 35.85 

N ° 19. S3 5.90 11.35 36.96 

Func t ion 

Research IS. 83 6.23 12.27 37.19 

Other 19.40 5.75 10.99 36.10 



ERIC 21 



REFERENCES 



Baker, E • L . and Herman, J.L. (1985). Educational evaluation: 

Emergent needs for research. Evaluati on Comment 7(2). 
Boyer, CM., Ewel 1 > P.T., Finney, J.E., and Mingle," J.R. (1987). 
Assessment and outcomes measurement : A view from the 
States. AAHE Bui letin 39(7) : 8-1S. 

Brown, m.K. (1986). Evaluation of nonacademic units, a new 

challenge for IR. Paper presented at the Southern Assoc iat ion 
for Institutional Research/Southern College and University 
Planners Annual Meeting, Pipestem, West Virginia, October 1*86. 

Council on Postsecondary Accreditation (Winter 1986). Education 
quality and accreditation: A call for diversitvtcontinui tv 
and innovation. Accreditation, The Quarter ly Newsletter of 
the Counc i 1 on Postsecondar y E ducat ion 1 1 ( 1 ) , l -5 . 

El-Khawas,E. (1987). Ca mpus Trends 1937. ACE, Division of ' 
Policy Analysis and Research: Washington, DC. 

Ewell, P.T. and Jones, D.P. Cno datel. The Costs of Assessment. 
Boulder, CO: NCHEMS, Contract no. 400-83-0009 ~ 

F eas 1 ev , C . E . (1 980 ) . Progra m Evalu ation, AAHE-ER I CJH i ah er, 
Educat ion R esearch Report No . 2 , 1980 . Wash i ng ton , DC : FRIC* 
Clearinghouse on Higher Education. 

Fincher, C. (1978). Importance of criteria for institutional 
goals. In Robert H. Fsnske (Ed.). Us ing goals in research 
and &ianninq( 1-15) . New Directions for Institutional Pp- 
search (19). San Francisco: Jossey-Bass. 

d E .EL 87 Higher Ed ucation Direct ory ' ( 19S 7) . Higher Education 
Pub 1 icat ions . 

Johnson, F.C. and Christal, M.E. (1985). Preparing for seli-studv. 

Tn g AIR Prof ess i onal Fi 1 e . No. 22. 
Lenning, O.T. (1986)* College students: Boiling it down. The 

Pen l Postseco ndary Educa t i on Network. Washington, DC: 

AERA, 5-6. 

Mcore, K.M. (1986). Assessment of institutional effectiveness. 

In John Losak (ed.), Appl yj no Insti tutio nal Re searc h in 

^ec^sion Makjjna. New Directions for Community Colleges\56: 

49-60. San Francisco: Jossey-Bass. 
Ih*. Raleigh News and Observer (September 6, 1987). Bennett 

wants college accreditation linked to what students learn. 

p. SSA. 

Pace, C.R. (1979). Measurin g outc omes of coUeqe: F i_ve ^ears 

iiH d 1 n 3 5 and recommendations for the future." Sat. Francisco: 
1979. * 

Reichard, D. (1987). Presidential address given at the Twentv- 
Seventh Association for Institutional Research Annual Forum, 
held in Kansas City* Missouri. 

Rogers, J. (1987). Accreditation: Its national and regional value 
in 1987. Paper presented at the National Association for 
Equal Opportunity in Higher Education's Twelfth National 
Conference on Blacks in Higher Education. Wash i ng ton , DC , 
April 9-1S, 1987. 



19 



Southern Association of Colleges and Schools. C riteri a for 

£Lccre^iiaJ.^onj_ Commissi on on Col le ges. Atlanta: S£CS. 
1987. 

Thrash, P.T. (1987). A report on the role of outcomes evaluation 
in accreditation process. North Cent ral Association 
Quar t er 1 y 61 ( <+ ) : ^81-^90. 

Troutt, W.E. (1979). Regional accreditation evaluative criteria 
and quality assurance. Jour nal of Hioher Education 50* 
199-210. ' ~ ' 

Van Maanen,J. (1987). Managing education better. Kevnotp 
address to the 27th Annual Forum of The' Association 
for Institutional Research, Kansas City, Missouri. 

Warren, J.R. < e d . ) . (1983). Meeting the new standards. In Ne-w 
Di£ec.tions for Higher Education. Ed. Martin Kramer. S a r, 
Francisco: Jossev-Bass, 1-7. 



20 



